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Abstract. We apply the replica analysis established by Gardner to the multi- 
constraint continuous knapsack problem, which is one of the linear programming 
problems and a most fundamental problem in the field of operations research (OR). 
For a large problem size, we analyse the space of solution and its volume, and 
estimate the optimal number of items to go into the knapsack as a function of the 
number of constraints. We study the stability of the replica symmetric (RS) solution 
and find that the RS calculation cannot estimate the optimal number of items in 
knapsack correctly if many constraints are required. In order to obtain a consistent 
solution in the RS region, we try the zero entropy approximation for this continuous 
solution space and get a stable solution within the RS ansatz. On the other hand, 
in replica symmetry breaking (RSB) region, the one step RSB solution is found by 
Parisi's scheme. It turns out that this problem is closely related to the problem 
of optimal storage capacity and of generalization by maximum stability rule of a 
spherical perceptron. 
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1. Introduction 

Recently, Korutcheva, Opper and Lopez Jl| pointed out that statistical mechanical 
analysis based on the replica calculation M can be used to investigate some specific 
statistical properties of optimal solutions for an optimization problem, the so- 
called knapsack problem. The knapsack problem they studied is one of the integer 
programming problems in which the variables are all integer. The problem is stated as 
follows. Let us suppose that a man climbs a mountain. He puts N items si, S2, • • •, sat 
in his knapsack. Each item has its own weight ai, 02, • • •, a^v and "utility" ci, C2, • • •, cat 
which means, for example, its own "price" or "necessity in climbing a mountain" . He 
cannot bring all the N items in his knapsack and he has to leave some of them. He 
must decide which combination of items is best for him. We can write down the above 
situation as 

{N N "] 

J=l J=l J 

where the decision variable Sj = 1 is defined such that the jth item goes into the 
knapsack, and Sj = otherwise. The constant h represents the weight limit. If we 
have to take account not only of the strength of the man but also of the capacity of 
the knapsack itself (i.e. the volume of the knapsack is limited), this new constraint 
must be considered and the problem becomes more complex. Thus it is meaningful 
for us to generalize our problem to a new form with K constraints as follows. 



{N N I 

X^'^J'^J I '^°-k]Sj<hk,k = 1, • • Sje{0, = 1, • • •,iV > (2) 

Following Korutcheva et al [0, we set the utilities Cj all 1/2, and we set the limits 
all representing capacities of the man, of his knapsack, etc, to h. 

In order to treat the multi-constraint (K^N) knapsack problem in a more familiar 
way for physicists, we convert the decision variables Sj — (1,0) into spin variables 
Sj = (1, —1), by transformation Sj = 2(sj — 1/2). The load Ukj is also transformed as 
Ofej = 1/2 + ^kjy where the quenched random variable ^kj has zero mean and variance 

and is assumed to obey the distribution 

As loads ttki, which means "weights" for example, should be positive, we must choose 
the above variance small enough and for this small variance, the fraction of items 
for which aki < for all k vanishes exponentially with increasing K . In this paper, we 
use (7 = 1/12 following Korutcheva et.al Using these transformations, our problem 
is rewritten as follows. 



(3) 



+ ^(W' k=l,--;K, sje{l,-l},j - 1, • • •, iV [ (4) 
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where 

1 ^ 1 ^ AT 

y({s}, {a,,}, b) = - ^(1 + + 4 E + T " ^ 

From the constraints appearmg in Eqs.(4) and (5), if we treat the case of N/A-^b, we 
can put most of the items into the knapsack. On the other hand, the number of items 
to be in the knapsack is too small when we set N/A':$>b. For this reason, we treat the 
case of N/A = b which means that about half of the items go into the knapsack. 

The case of finite K in the limit N infinity was investigated by Meanti et al |^ 
by the Lagrangian relaxation technique. They gave the explicit results for the special 
cases oi K — 1 and K = 2. Recently, Fontanari presented an explicit calculation of 
the annealed entropy of the configuration s* which satisfies K constraints and gives 
the total benefit E = s*/2. He estimated the upper limit of the total benefit using 
the zero entropy condition. 

He compared it with the exact result from Lagrangian relaxation for the case of 
if = 1,2 (exact solutions) and K > 2 to conclude that the annealed approximation 
becomes good as K increases. 

The original knapsack problem consists of only one constraint, or at most, several 
constraints. One may feel that only such a case is worth investigating. However, in 
actual operations research field, we sometimes face multi-constraint [K^N) knapsack 
problems jsj, an example of which is investigated in this paper. 

If we look at the above problem from an actual operations research (OR) point 
of view, difficulties lie in the discreteness of knapsack variables. When hardship of 
this sorts confronts us, it is customary to apply the "linear programing relaxation" 
technique Linear programing relaxation is an approach to solve the integer 

programing problem approximately, which brings us to the neighborhood of the exact 
solution. Actually, for linear programing problems, a lot of useful methods, for 
example, the simplex method or the interior point method, have been proposed and 
improved 

In addition, it is worth while to investigate the objective function for a linear 
programing problem itself (not as a relaxation of integer programing problem such as 
the orignal knapsack problem) . Linear programing problem is defined as follows 

1 

'^akjSj<b,k^ I., - ■ ■,K,0<s.j<oo,j = l,---,iV > (6) 
j=i J 

Problems of the above style occurs very often in our life. "The diet (nutrition) 
problem" is one of them. The diet problem ask us to determine the intake {sj} of each 
food. Each food Sj has its own nutrients akj, i.e, calcium, protein, vitamin, energy, 
etc., and one has to take each nutrient within the limit, i.e akj<b (fc = 1, • • •, K), 
where akj is kth nutrient of the jth food. We should notice that valuables in this 
problem are real numbers, not integers. And linear programing problem has more 
application than for integer programing problem Q. For two reason, it is worth 
investigating the multi-constraint (K^N) continuous knapsack problem. 

When we look at our knapsack problem from the stand-point of linear programing 
relaxation, the knapsack valuable Si, which has a real value and determines whether the 
ith item goes into knapsack, takes any value in the real subspace satisfies J2i — ^- 
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We treat the next real-variable problem in the present paper. 

N I ^ I 

max <([/=- + - ^ I r({s}, {ak^}, b)<0, k^l,--;K, s,e{~(x, +00}, j = 1, • • •, TV J> (7) 

and Eq. (5). Although this relaxation of constraint ~ ^ ^^"^ SiG{— cxd, +00} 

in Eq.(7) somewhat obscures the direct significance of the variable Si, the global 
macroscopic behavior of the system is expected not to be affected by this 
approximation as was mentioned above. And this constraint could be justified as 
simply the spherical version of the problem investigated by Korutcheva et al |Q] 

Using this linear programing relaxation technique, we can obtain the approximate 
configuration s* = (s*, • • •, s^) and we get higher value of the objective function (or 
the total profit) 



N 

I 

E 



1 ^ 

In this paper, we investigate the knapsack problem by the linear programing relaxation 
technique and estimate the objective function when the number of constraints is given 
for a large problem size. 



2. Replica symmetric theory 

For the parameter regions mentioned in the previous section, we consider the case 

N 

with continuous variables under the normalization constraint ^^(sj)^ — N. For a 

large problem size, the fractional volume of solution V{M, {£,ki}) to Eq. (7) is written 
by introducing the "fluctuation of the magnetization" around 0, which is of order 
Oil), 



M = ' ' 



1 ^ 



as 



N . , 

2 — 1 



V{MACk^}) - — '—^ — ^jT^ , ^ ^^(10) 





n Jds.slj2( 

k=l \i=l 



This expression is similar to Gardner's volume of a binary spherical perceptron with 
local field — ^^Si^fci/V^iV and stability criterion constant —M/2 appearing in the 

i 

problem of optimal capacity]^ |^ or of the generalization of maximum stability rule 
Thus it is possible to use the same technique. And this V is the value of configuration 
{si} which have a fixed M , implying also a fixed utility U . The typical value of V is 
given by exp(<Clogl^:^), where <C- • denotes the averaging over quenched disorder. 
We perform the average of logV with the distribution (3) over the different sets of 



solutions using the replica trick as follows. 
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«;logy> = lim 



<T^"> - 1 



n— ►O n 



(11) 



The averaging of the power <^V"^ is accomplished by introducing an ensemble of n 
identical replicas 



N 



f ) U J2^sn' -n\s[ vnm - Y^s- 



n / '^"-^"^ ( E^""*' 

k=l'' \<=1 



f-N 



Then we introduce the order parameter which is the overlap between two solutions 
labeled by two replica indices a and /?, 



By the standard procedure, we obtain within the replica symmetric ansatz 



= exp 



Nn |cxtr^_^_. ,^G(g, E, q, M) + 0{j-) 



1 



(13) 



(14) 



where extr^ e q m denotes the extremization with respect to the parameters q, E, q 
and M. Here 



G = aGi + G2- -qq + iE 



(15) 



->(12) 



Gi=log TT / / dxaxexp 



E(^«)^ ~ ^^(1 + 9) E E ^"^/^ 

a a /3 a</3 



(16) 



Go = 



/OO 
n 
-OO „, 



(is"exp 



-zM ^ s" - i« ^ ^ s'^s^ - iE ^(s")2 

a a<f3 a 



(17) 



Using the saddle point equation with respect to M, we obtain 

M = (18) 
Using this result and saddle point equation with respect to E, we get 



GiM,q) = al Dt logg ( ^/(^-j_VT + ^ )+llog(l-g)+l^(19) 



where 



27r 



21-q 



(20) 
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and 

/"OO 

H{x)= / Dt (21) 

J X 

Finally we estimate the saddle point of Eq.(19) with respect to q in the limit q^l. 
This means that only one optimal solution (which satisfies all K = aN conditions in 
Eq.(7)) is selected for a given number of constraints in this limit and then optimal 
Mopt can be calculated by q^l for this optimal s which satisfies all K constraints (7). 
Thus we obtain M^pt as a function of a = K/N. This result is shown in figure 1. 

From this result, we see that M — a lines show the next behavior: As the 
number of constraints increases, M decreases and as a result, total utilities decrease 
(we should remember that total utilities are given as U ^ N/4 + J2j=i Sj/2 and 

AI = Si/\fN). In figure 2, we also plotted the optimal profit Mopt for the case 

of Ising variables using the zero entropy condition of solution discrete space taken 
from [0. This result shows that we can obtain the larger optimal profit than that of 
the original knapsack problem by our relaxation of variables. 



3. Three relevant lines 

3.1. AT line 

In order to investigate the local stability of the RS solution, we follow the usual 
Almeida and Thouless ||l^ argument and introduce the fluctuations of the order 
parameters around the RS order parameter as follows. 

qaf} = q + 6qal3 

= q + Sqap 

Ma = M + 5Ma (22) 

M„ = M + 5Ma 

Ely — E SErv 



It turns out that only fluctuations in q^,^ and q^fs lead to instability. The function to 
be investigated has the form 

aGiiqa, Mo,) + G2iq»f3, Ma, Ea) + 

qal3qaf3 

P a<fj 



(23) 



From this expression, we can apply Gardner's |6|||7| analysis to our case and obtained 
AT-fine as 



r°° { L z'^Dz ( \, zD 
loo I /r Dz \ /, D, 



< 1 



where 



I. 



(24) 



(25) 



/(M/(2(T)-VT+^t)/\/T^ 

and q is the order parameter obtained by the replica symmetric calculation and is 
given as 



q = a 



jDt^^^q-2t 
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X 



1 



exp 



( 



{Ml {2a) - t./TTq)^ 
2(1-9) 



) 



/ 



H 



M/{2(j) - t^/TTq 




We plot this condition Eq.(24) in figure 1. 

From this figure, we see that the replica symmetry is stable for M > Mat — 0.100 
(a < aAT — 0.846). However, when M becomes smaller than this critical value, the 
replica symmetry breaking occurs and the replica symmetric saddle point becomes 
unstable. The AT point {aAT{MAT)) is a critical point where the replica symmetric 
order parameter qfis = Q splits into (one step) replica symmetry breaking saddle point 
represented by qo and qi. If this RSB transition at aAT is continuous, the AT condition 
Eq.(24) gives a correct criterion of replica symmetry breaking. On the other hand, if 
the RSB transition is discontinuous (first order), we should regard this discontinuity 
point as the symmetry breaking point rather than the AT stability limit. Fortunately, 
as we see in the next section, this transition is continuous and the AT argument is 
correct. 

In order to understand this situation physically, we can investigate the replica 
symmetry breaking from the dis-connectivity of the solution space s according to 
Monasson and O'Kane |Q, who treated RSB of perceptron with non-monotonic 
transfer function. 

The solution condition Eq.(7) can be rewritten as 



Then if M is positive, for many constraints K, the space of s which satisfies Eq.(7) as 
well as the normalization constraint — ^ consists of a single domain and the 

solution space is connected. On the other hand, as M becomes negative, the space of 
s splits into a lot of domains and the solution space is disconnected. 

Here we can estimate the 0(1) term roughly appearing on the right hand side of 
the above inequality as ^j'/Z/V^ (1///V)x ViVa = 1/12. We should notice that 
this term is of 0(1) and we used the standard deviation cr = 1/12. Therefore if we 
investigate the dis-connectivity of solution space, replica symmetry must be broken 
for M < 1/6 = 0.1667. This value is not so far from the value obtained by the AT 
argument. We conclude that this dis-connectivity leads to replica symmetry breaking. 
For this RSB region, we will later find the one step replica symmetry breaking solution 
following the scheme of Parisi 

3.2. Zero entropy line 

In the previous subsection, we calculated the volume of solution space exp(log^F^) 
for quenched random loads ^ki- For optimal Mgpt, this volume shrinks to zero 
continuously. 

Thus the increase in the number of constraints leads to a decrease of the solution 
space s in a discrete manner, and the entropy of solution space is S{Mopt) = at 
M = Mopf Following the idea suggested by Krauth and Mezard we expect that 
the optimal M for continuous variables may be obtained when the volume of solution 
contains typically a single point of hypercube. From analogy with the discrete- variable 
problem, this should occur around 




(27) 




(28) 



3.2.1. Annealed calculation We first calculate the condition Eq.(28) by the annealed 
approximation as 

log<y> = G-H = -log2 (29) 
This condition is easily calculated by 

G = alogi/(^^) +i(l + log(2^)) (30) 



H=^(l + log(2^)) (31) 



Finally we get 



alogi?!^^) =-log2 (32) 



This is shown in figure 2. 



3.2.2. Quenched calculation Next we calculate the above zero entropy condition 
Eq.(28) by the quenched calculation defined as 

«logl/» = ^ (33) 

This leads to 

./ Dt logi/ ( W2a-^^ ^) ^ 1 _^ ^ _ ^) ^ _j^g2(34) 

We plotted this result also in Figure 2. 

From figure 2 we find that this line Eq. (34) obtained by the quenched zero entropy 
condition is stable in the RS region. In figure 4 we plotted the behavior of the order 
parameter g as a function of a which gives zero entropy and satisfies Eq.(34). From 
these results, we may conclude that our zero entropy approach gives a meaningful and 
consistent criterion for the optimal Mopt. 

3.3. One step RSB solution 

In this subsection, we calculate the one step replica symmetry breaking solution 
following Parisi Q . We try to find a first candidate for matrices q and q appearing 
in the free energy. Using Parisi's suggestion, we divide the n rephcas into m/n groups 
of m replicas. Next we set qaj3 = qi, qaf3 = <?i, if a and (3 belong to the same group, 
and qaf} = qo, qap = Qoi if and (3 belong to different groups and we set qaa = 0. We 
express q and q in terms of a tensor product, 

g = (91 - 9o)in/m'8)e„ie^ + goe„e^ (35) 



(91 - qo)^n/va<i 



(36) 
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where is a fc-dimensional unit matrix and = {!,■ ■ -,1). Using this form of broken 
symmetry of repHca, we get the free energy as follows, 



hRSB{qi,qo,m) = ^ J Dylog J Dzl^H 



f M/{2a) - yVTTq^ - z^qy - qo 



+ ;r^log[l -qi+ m{qi - qo)] + - -)log(l - qi) + -j- ^3 

2m 2 m 2[1 — qi + m(qi — qo)] 

where we used the saddle point equations with respect to qi, qo, E and M. 

Before calculating the one step RSB solution, we had better investigate whether 
the one step RSB solution actually exists or not by minimizing the one step RSB free 
energy Jirsb with respect to the order parameters qo , qi and m. For example, we 
try to investigate the case of M^p^ = 0. Varying a in the free energy fmsB, we look 
for the set of order parameters qo, and m to minimize fiRSB- In table 1, we list the 
result for the typical value of a. Prom this table, we see that until a'^aAT = 1-20, 
the replica symmetric saddle point (which is qo — qi — q and is independent of m) 
is stable and the one step RSB solution does not exist. When we exceed the critical 
value a AT, the replica symmetric saddle point becomes unstable and order parameter 
q splits into one step RSB order parameters qo and qo continuously as shown also in 
figure 3. From this continuous transition, we conclude that the AT argument in the 
previous section is consistent. Comparing the free energy f^s and Jirsb, we see that 
the global minimum of this system goes from the RS saddle point q to the one step 
RSB saddle point {qo,qi) at the AT instability point. Therefore, the one step RSB 
solution actually exists. 

In order to obtain the solution in the limit g'l— >1, we use the scaling qi = l — e,qo = 
Q and m = jie to get 

FiRSB{Q,l^,e) = ^/iijsB(Q,M) +0(loge) (38) 



Here 



where 



and 



a f 1 
hRSB{Q,lA = - / Dy\og[Hy+H2]+—\og[l+p,{l-Q)] + 



Q 



2[1 + m(1-Q)] 




^ '_H Z!—^ \ ' ^ (41) 

The saddle point equations with respect to \i, Q and e 



OF 

9/i 



dF 
dQ 



= F 



= (42) 



give the critical values ac calculated by the one step RSB scheme. We solved Eq.(42) 
numerically and show the result in figure 2. 
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From this figure, we see that the replica symmetry breaking decreases the optimal 
Mopt slightly. We also confirm this result from the fact that replica symmetry breaking 
point q {a at) comes close to 1. In figure 5, we also show the order parameter qo = Q 
as a function of the optimal profit Mopt- From this figure we see that, as Mopt becomes 
close to 0.100, i.e. as replica symmetry breaking becomes weak, Q becomes 1 gradually 
{Q = Qo = 0.9912 for m = 0.100). This result is reasonable from the meaning of replica 
symmetry breaking. The reason is that in critical limit of a, the one step RSB order 
parameter go and qi becomes close to Qbs = Q a-nd 9— »! if replica symmetry docs not 
break down. 

In order to confirm that this one step RSB solution is exact, we must check the 
stability of the one step RSB saddle point by the same technique as that used for 
the AT line and investigate if the free energy of one step calculation is lower than 
that of the second step calculation. Or the distribution of sizes of the "disconnected" 
domains of solution space s must be computed analytically by the same technique 
as in Monasson and O'Kane ||l^. This is a highly non-trivial problem. However we 
can conjecture that the two step replica symmetry breaking can hardly decrease the 
optimal profit even if two step RSB solution exists, because the optimal profit of replica 
symmetric calculation and that of one step replica symmetry breaking calculation are 
very close to each other as we saw above. 



4. Summary and Discussion 

We have calculated the optimal profit Mopt explicitly for the case of continuous knap- 
sack variables and the number of constraints K is of the same order with the number 
of items N by replica symmetric calculation explicitly. From the AT argument, the 
replica symmetric solution becomes unstable for M < 0.100. We also investigated this 
symmetry breaking point by minimizing the one step replica symmetry breaking free 
energy directly and saw that this transition is continuous and the AT line is valid. 
For the argument of the problem of the optimal storage capacity of the perceptron 
with non-monotonic transfer function investigated by Monasson and O'Kane ||l^, the 
physical meaning of this RSB can be understood as the dis-connectivity of the so- 
lution space. Therefore, from the condition of dis-connectivity of solution space, we 
roughly estimated where replica symmetry breaking begins. For the RSB region, it 
was necessary to try to find a replica symmetry breaking solution according to Parisi's 
procedure Q Q. Thus using Parisi's scheme, we obtained the one step RSB solu- 
tion and the result shows that replica symmetry breaking makes the Mgpt decrease 
slightly. Because it is not easy to confirm that this one step RSB calculation is exact, 
we can roughly expect that even if we calculate the two step RSB solution or three 
step RSB solution or further step RSB solution, the optimal profit obtained at these 
further RSB saddle point should hardly increase. In stead of investigation in this 
direction, in the replica symmetric region, we can find a consistent solution using the 
zero entropy approach and this result agrees with the discrete variables case [|l| for 
the most part. This result may also support the validity of their RS calculation. The 
present statistical mechanical approach is expected to be applicable the other linear 
programming problems. 
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Figure captions 



Figure 1 . Mopt — ct line calculated by the replica symmetric theory, AT line and one 

step replica symmetry breaking solution. In each line, we set cr = 1/12. The replica 
symmetric solution is stable above M~ 0.100. In this figure, the region where the 
replica symmetry is broken is shown enlarged. The optimal profit Mgpt decreases 
slightly by the one step replica symmetry breaking calculation. 



Figure 2. The zero entropy lines (quenched and annealed calculations). The 
quenched zero entropy line is stable in the RS region. The optimal profit for Ising 
variables is also plotted. This line is very close to the quenched zero entropy line. 



Figure 3. Order parameter g as a function of the relative number of constraints a 
for the case of Mopt = 0. Replica symmetric order parameter q splits into the one 
step replica symmetry breaJjing order parameter {qo,qi) continuously at a at ~ 1.20. 



Figure 4. Overlap q which satisfies the zero entropy condition. 
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Figure 5. One step RSB order parameter qo = Q as a function of tlie optimal profit 
Mopt in tlie limit of gi— >1. As replica symmetry breaking becomes weaJj ( as Mopt 
closes to 0.10, a{Mopt = 0.100) = oat)) Q closes to 1. 



Tables and table captions 



a 


Qrs 




qi 


m 


Irs 


Irsb 


0.90 


0.7129 








-1.5336 












1.00 


0.7773 








-1.9346 












1.10 


0.8408 








-2.4859 












1.20 


0.89U1 








-3.3794 












1.25 


0.9073 


0.8213 


0.9901 


0.1688 


-3.7642 


-5.7847 


1.30 


0.9211 


0.8155 


0.9902 


0.1858 


-4.4110 


-6.0763 



Table 1 . The replica symmetric order parameter q and the 1-step replica symmetry 
breaking order parameters ((jo,gi,m) are listed with the free energies for several 
values of a. Untill a^aj^x = 1.20 the RS saddle point is stable and the 1-step RSB 
solution does not exist. Above the a^T, the free energy of 1-step RSB approximation 
is lower than that of RS approximation. 
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